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Abstract: This paper attempts to examine different methods of content analysis for 
student inline discussion that takes place on bulletin boards. The purpose of that 
analysis is to determine whether higher order thinking can be distinguished wthin 
transcripts of dialogue. The context of the analysis was a higher education 
undergraduate course. A group of students were presented with a discussion topic and 
were given a series of criteria against which their work would be assessed. At the end 
of the discussion their dialogue was analysed using three different techniques which 
were then compared to identify whether any one method could be recommended to 
other practitioners. Suggestions for evaluation of bulletin board transcripts are made 
on the basis of this investigation. 
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^ The development of discussion boards as learning tools 

Discussion boards or bulletin boards are part of a general category of CMC. Discussion boards allow 
students to interact with one another and with a facilitator and can be used for a variety of teaching 
techniques designed to stimulate cognitive and metacognitive skills. 

An extensive literature base (Harasim, 1990; Mason, 1994) has developed which supports the educational 
value of CMC as a teaching and learning strategy. However, most investigations into the benefits of this 
form of interaction have tended to be quantitative in nature (McKenzie & Murphy, 2000; Nastasi & 
Clements, 1992) concentrating on either the number of exchanges that take place, or on diagrammatic 
representations of the pattern of interaction between participants. During the last decade however, methods 
of analysis have been applied to this communication technique to identify more precisely the varied 
educational dimensions that may be found within the online text (Henri, 1992; Gunwardena et al, 1997). 
Different forms of ‘content analysis' have been developed and refined to aid the categorisation of dialogue 
produced in any debate or online seminar. MacKinnon and Aylward, (2000) for example, have used a 
system of coding online dialogue using ‘cognotes', where the e-moderator attaches icons to ongoing text in 
order to indicate the nature of the interactions which are occurring. This is intended to encourage higher 
student achievement and more positive relationships between the participants. 



Aim of study 

Currently, content analysis has been seen as a time-consuming activity which, while it can produce 
valuable insights for educational researchers, does not lend itself to practical application by teachers who 
might want to evaluate the online discussion element of their programmes. This pilot study seeks to 
compare the relative strengths and limitations of established content analysis techniques and then propose a 
simp ler method that is useful and practical. This would then allow course providers to decide whether their 
course participants are displaying cognitive and metacognitive aspects of learning at the level they expect. 




Depth of learning 

McLoughlin and Oliver (1998) have indicated that Higher Order Thinking (HOT) is not easy to define but 
can be recognised when encountered. These authors have summarised recent debate on the definition of 



HOT as referring to thinking which is complex, multi-faceted, self-directed and one in which the learner 
plays an active role (1998, p. 242). This is in line with Sternberg’s (1985) theory of intelligence that 
considers the basis for higher order cognitive processing to be executive processes used in planning, 
monitoring and decision making. For the purpose of this paper cognitive skills are generally regarded as 
applying to student understanding, reasoning, and the development of critical thinking and problem solving 
skills, while metacognition relates to knowledge of one’s own cognition and the further regulation of that 
cognition (Kara, Bonk and Angeli, 2000). Henri and Parer (1992), in addition to their model of content 
analysis, also identified the level of information processing as ‘surface’ or ‘in-depth’. They recognised 
surface processing to include: making judgements without justification; stating that one shares opinions 
already stated; repetition; and irrelevant questions. In-depth processing, Henri stated, was demonstrated 
when participants linked facts and ideas; offered new elements of information; discussed advantages and 
disadvantages; and made justified judgements. 



The study 

A group of 10 adult learners on a higher education programme took part in an online discussion which was 
an obligatory part of their course assessment. None of these participants had previously experienced online 
discussion. The asynchronous computer conferencing software used was the discussion board contained 
within BlackBoard Courseinfoy a web course management system. The discussion was entitled: 
“Integrating technology into education - a constructivist approach”. The students were required to take part 
on at least four occasions in a discussion on the topic within a four-week time frame. The criteria upon 
which their contributions were assessed in the course were obtained from Campbell’s experiences (1999) 
and were: regularity of contribution, links to the comments of others, sharing of ideas and experiences, and 
reference to readings or Websites. 



Evaluation by content analysis 

Content analysis involves comparing, contrasting and categorising elements of written dialogue for 
meaning, as opposed to discourse analysis which examines the process of communication including 
specific speech acts. A range of examples of models have been selected for this study including those 
developed by Henri (1992) and Gunwardena, Lowe and Anderson (1997) and an earlier method of 
determining the complexity of student responses, the Biggs’ SOLO taxonomy (1982). Modifications to 
these schemes are also incorporated in the analysis, ‘since every computer conference will have its own 
unique attributes, researchers may have to design electronic discussion group evaluation criteria on a case 
by case basis,’ (Hara, Bonk and Angeli, 2000, p.l43). In particular the models were chosen to enable higher 
orders of thinking to be identified in the dialogue, as this is frequently identified as a required outcome in 
tertiary level education programmes. 

Table 1: Gunawardena, Lowe and Anderson content analysis model (1997) 



Categories For Data Analysis 


Examples 


Phase 1 : Sharing and Comparing 


Observations, examples and descriptions, basic agreement 


Phase 2: Discovery and exploration of 
difference 


Questions re differences, clarifying statements 


Phase 3: Negotiation of meaning and co- 
construction of knowledge 


Identification of common ground, statements of compromise 


Phase 4: Testing and revision of ideas 


Testing of ideas against personal knowledge 


Phase 5: Awareness of newly constructed 


Metacognitive statements, reflection 



Model 1: Gunawardena, Lowe and Anderson’s (1997) criteria for content analysis 
This model was originally designed to examine the construction of knowledge within online debates in a 
similar manner to the design of Nastasi and Clements (1992) who considered that the resolution of 
cognitive conflict could be used as a measure of cognitive learning. See table 1. 
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Model 2: Henri and Parer *s qualitative criteria for content analysis (1992) 

This model identifies five dimensions within the framework of the content analysis: participative, 
(examined by quantitative analysis), social, interactive, cognitive and, metacognitive. Each of these have 
operational definitions and related indicators, however only the latter two have been examined in this study 
as the focus of the study was to analyse participants’ learning outcomes rather than the collaborative 
process of achieving that result. Nevertheless, it is recognised that in a constructivist learning environment, 
where collaborative learning is occurring in a social setting, interactions between the participants must play 
a significant role in the achievement of understanding. 



Categories for data analysis 


Definitions/Indicators 


Cognition 




1 . Elementary classification 


Observing a problem and its linkages, questions, 
basic descriptions 


2. In-depth classification 


Analysing a problem re beliefs and assumptions, 
referential criteria 


3. Inferencing 


Inducting and deducting (based on prior 
propositions), generalisations, conclusions 


4. Judgment 


Making decisions (1 agree), appreciations, criticisms 


5. Strategy application 


Proposing actions, evaluations, decisions 


Metacognition 




1 . Metacognitive knowledge 


Perceptions of oneself as learner and thinker, 
realisation of strategic knowledge 


2. Metacognitive skills 


Planning, regulation, evaluation and self-awareness 



Table 2 : The Henri and Parer content analysis model 



Kara, Bonk and Angeli (2000) have added the categories ‘reflection’ and ‘self-questioning’ to Henri’s list 
of metacognitive skills and these have been applied in this study. Even with these modifications, 
metacognition is not often clear, possibly because it occurs infrequently in the discussion being studied. 

Model 3 - Biggses SOLO taxonomy (Biggs and Collis, 1982) 

This form of textural analysis, designed primarily for the print-based word, was an earlier attempt to 
examine the complexity of students’ responses. Although it was not designed to function as a model of 
content analysis, it was trialed in this study to see if Biggs’s categories of HOT would provide a useful 
alternative technique. The outcomes have the advantage of being generic, and provide a fairly clear cut-off 
between students’ surface and deep learning approaches, at a point separating level three from level four. 



Level of outcome 


Indicators 


1. Prestructural 


Irrelevant information, no meaningful response 


2. Unistructural 


Answer focuses on 1 relevant aspect only 


3. Multistructural 


Several relevant ideas, but not co-ordinated 


4. Relational 


Integrated, meaning is understood 


5. Extended abstract 


Answer goes beyond information given. 



Table 3: Levels of Biggs’s SOLO taxonomy 



Method applied in this study 

The fifty messages contributed by students, primarily in chronological order, were first listed by dialogue 
number. Remarks made by the e-moderator were deleted from this list. Initial entries placed on the board 
during the first face-to-face session were also removed as these did not pertain to the assignment but were 
designed to familiarise students with the process of contributing. 

Following this preparatory stage the three main techniques described above were applied to the whole 
sequence at an entire dialogue message level. In addition, surface processing compared to in-depth 
processing following the Henri model was also evaluated. Within each message the category was selected 
on the basis of the highest order of thinking demonstrated within the dialogue. In other words, messages 
were not sub-divided into smaller units of meaning. There were three reasons for this; firstly, subdivision 
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would have simply produced additional data indicating an increased amount of lower value learning levels 
without affecting the extent to which higher order thinking might be discerned. Secondly, this whole 
message analysis overcame the problem of deciding where to slice the dialogue to achieve appropriate 
subsections. Thirdly, it saved a large amount of analytical time. It did however obscure the precise 
identification of student performance, as described in Gunawardena’s model where each of the five phases 
is categorised in more detail. To increase the rigor of this subjective process a second researcher examined 
every fifth entry and applied the same series of content analyses to this 20% sample. 



Results 

All models showed a distribution of rating scores from 1 - 5, with relatively few responses at the higher 
level of cognition. Table 4, indicating the relative ratings allocated by each model, shows that the 
Gunawardena model produced a greater proportion of low-level ratings than the other two. In the Henri 
model metacognitive references were also listed (M) which occurred mainly in the cognition category 5 
ratings but also one occasion in the category 4. This table indicates overall comparative frequencies but 
does not indicate similarities or differences between the models per individual dialogue. 



Content analysis 
rating 


Gunawardena 


Henri 


Biggs 


1 


24 


15 


6 


2 


9 


16 


8 


3 


9 


8 


16 


4 


6 


5 + IM 


15 


5 


2 


6 + 3M 


5 



Table 4; Content analysis rating scores for all three models 
To examine whether the three models did show consistency between the individual ratings a simple 
correlation analysis was applied to show degrees of association or similarities between them all (see Table 
5). 





Gunawardena 


Henri 


Gunawardena 


- 


- 


Henri 


.7405 


- 


Misgs 


.6624 


.7025 



Table 5; Correlation Coefficient between content analysis models 

Table 5 shows that there is a strong correlation between the Henri and Gunawardena models and a lower 
one between the Gunawardena and the Biggs analyses. The reason for this difference is that results from 
the Gunawardena rating allocate 50% extra scores at the lower end of the learning level scale in comparison 
to Biggs (see Table 4). Nevertheless, all three models do show statistically strong degrees of correlation. 



Conclusions 

Two main conclusions can be dra\vn from this study. Firstly, practitioners wanting to apply one of these 
models will be able to do so quite readily if they select their categories at the level of the whole message, 
rather than sub -categorising the text. The consistent application of any appropriate technique will then 
indicate levels of HOT in their own online discussions which can then be used for modifying teaching 
techniques. 

Secondly, the nature of the discussion and the expectations of the course providers should determine the 
choice of the primary model selected. If the aim is to encourage a collaborative learning environment, then 
the Gunawardena model provides more insight into whether or not that has been achieved. This model also 
incoqDorates metacognition into the main framework, which is essential if any evaluation of HOT is 
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required. The Henri model, with its metacognitive aspect applied as a separate framework dimension tends 
to highlight individuals’ success in displaying internal cognition in a more Piagetian sense, although within 
some of her categories there are opportunities to acknowledge the interactive nature of the discussions. A 
major criticism of Henri’s model (McLoughlin and Luca, 1999, p.223) is that ‘it was designed for contexts 
where there was a strong teacher presence, and is not readily applicable to a learner-centred conferencing 
environment’. These authors consider that the Gunwardena model a more suitable tool for reflecting 
collaborative and social factors, on the basis that it proposes a social constructivist approach to knowledge 
building in a online environment. 

In the Biggs model the categories are clearly hierarchical which leads to an overall value judgement about 
the content of each message. This rules out completely the possibility that socially interactive functions are 
acknowledged as part of cognitive development. Although all three are structured to indicate levels of 
learning and deeper understanding demonstrated by the text, it is only the Gunawardena cognition model 
that relates this achievement to the degree of collaboration demonstrated. 

In considering what model to apply and what modifications might be required, the nature of the student 
body should also be taken into account. The learning style of participants is one issue that should be 
considered and ‘it is suggested that current research literature in the area of learning styles and strategies 
can provide instructional designers with insights into differences in learning and performance that can be 
factored into the design process’ (McLoughlin, 1999, p. 223). Research has shown (Klemm and Snell, 
1996) that unless discussions are summatively assessed, dialogue will be minimal. It might be asked if 
instructional designers are merely providing increasingly elegant hoops for learners to jump through. 



Summary 

The significance of this paper is that it should encourage online teachers to develop more effective teaching 
techniques by enabling them to evaluate the effects that their own online practices play in students’ 
cognition and metacognition. A summary of the analysis approach for online transcripts can be summarised 
in there steps, and requires the selection of a content analysis approach that is congruent with the intended 
learning outcomes and pedagogy of the course of study. 

The three stages for analysis are as follows: 

1. First, analyse dialogue content at whole message level. 

2. Second, allocate a rating according to highest level of thinking evidenced in message (See for example 
the scheme proposed by McLoughlin & Oliver, 1998). 

3. Third, select or modify a content analysis technique that reflects the philosophy of the course in respect 
of student learning outcomes and pedagogy. 

Data analysis and coding of transcripts can then be carried out to determine the forms of thinking and 
cognitive skills that are embedded in the dialogue. 
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